feat(automation): route trusted review-agent mentions - #787
Conversation
|
Warning Review limit reached
Next review available in: 4 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (15)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
|
@opencode-agent @cwl-noema-review @coderabbitai review Review exact head |
|
|
Pull request was converted to draft
seonghobae
left a comment
seonghobae
left a comment
There was a problem hiding this comment.
Exact-head reliability finding (valid, blocking the claimed receipt-based idempotency): dispatch_request() writes its only durable receipt after dispatching Noema, dispatching OpenCode, adding a reaction, and then posting the acknowledgement. If any later call fails after an earlier repository dispatch succeeds—for example, Noema succeeds and OpenCode or the reaction/comment call fails—the source comment remains unreceipted. The five-minute sweep retries it and dispatches the already-successful agent again. The current Noema repository-dispatch concurrency key is PR-scoped with cancel-in-progress: true, so this failure window can cancel/restart a legitimate review rather than merely producing a harmless duplicate.
Please fix test-first with fault injection at every external-call boundary. The durable protocol must track completion per requested agent and allow retrying only missing work. Because there is still an unavoidable crash window between a successful GitHub repository-dispatch response and recording that success, the downstream Noema/OpenCode entrypoints should also enforce idempotency using a deterministic key containing at least target_repository, PR number, exact head SHA, requested agent, and source_comment_id. Receipt/reaction failure must not cause completed agent work to be redispatched, and a partially completed request must not be treated as fully processed. Update the operator record and claim boundary accordingly; do not weaken the current exact-head or review-only controls.
seonghobae
left a comment
seonghobae
left a comment
There was a problem hiding this comment.
Exact-head blocking defect in the sibling-repository sweep: processed_comment_ids() accepts receipt markers only when the comment author is exactly github-actions[bot], but the scheduled sweep posts target-repository acknowledgements through TARGET_REPOSITORY_TOKEN (PR_REVIEW_MERGE_TOKEN, OPENCODE_APPROVE_TOKEN, or an OpenCode installation token). Those comments are authored by the PAT user or GitHub App bot—not by github-actions[bot]. Consequently, the next five-minute sweep does not recognize its own sibling-repository receipt and redispatches the same mention repeatedly for the entire lookback window (subject only to each run's dispatch cap). This also amplifies the partial-failure idempotency defect from my preceding review.
Please add a failing end-to-end sweep regression using realistic PAT and installation-bot acknowledgement identities, then bind receipt recognition to the authenticated target-token identity (or another durable, non-spoofable central receipt protocol) rather than a hard-coded local-workflow actor. Do not broadly trust arbitrary bot/user markers: a repository workflow or collaborator must not be able to forge a receipt and suppress a required review. Verify local github-actions[bot], cross-repository PAT, and installation-token paths, repeated sweep runs, mixed legitimate/forged markers, token rotation, and partial agent completion. The current head should not merge while the advertised organization sweep can create an unbounded redispatch loop.
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current headcf4400f7f8fe673f1f89c467a6629bd92dcaae2c. -
Head SHA:
cf4400f7f8fe673f1f89c467a6629bd92dcaae2c -
Workflow run: 31012190358
-
Workflow attempt: 1
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow (2 files)"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow (2 files)"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file: CHANGELOG.md"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
R2 --> V2["required checks"]
Evidence --> S3["Docs: review-agent-comment-invocation.md"]
S3 --> I3["operator or user guidance"]
I3 --> R3["Review risk: Docs: review-agent-comment-invocation.md"]
R3 --> V3["docs review"]
Evidence --> S4["CI script (2 files)"]
S4 --> I4["review and security gate shell path"]
I4 --> R4["Review risk: CI script (2 files)"]
R4 --> V4["bash -n plus Strix self-test"]
Evidence --> S5["Test (3 files)"]
S5 --> I5["regression suite"]
I5 --> R5["Review risk: Test (3 files)"]
R5 --> V5["targeted test run"]
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Stale-head OpenCode coverage blocker resolved on exact head 3bef8ad. Agent Mention Router Quality CI run 31077236939 passed 45 tests with 100% statement coverage, 100% branch coverage, and 100% public-docstring coverage. The current head also removes all transient PR-specific branch writers and binds both wrappers to the canonical payload digest. A fresh exact-head independent verdict remains required; this dismissal does not count as approval.
|
@opencode-agent @cwl-noema-review @coderabbitai review Please review exact head |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
set -euo pipefail
repo="ContextualWisdomLab/.github"
base="f070c504c1cb06891b800d7ab0cf6ac7d3cf8eae"
head="92c83f44c75cae42e90dc6984dc89c696b6c3db9"
printf '%s\n' '== Revision verification =='
printf 'checkout HEAD: '; git rev-parse HEAD
printf 'requested HEAD: '; git rev-parse "${head}^{commit}"
printf 'base: '; git rev-parse "${base}^{commit}"
printf 'merge base: '; git merge-base HEAD "$base"
printf '%s\n' '== Changed files =='
git diff --name-status "${base}..${head}"
printf '%s\n' '== Relevant files =='
fd -t f . .github/workflows scripts/ci tests docs 2>/dev/null |
rg '(agent[-_]mention|review-agent-comment-invocation|pr_review_fix_scheduler)' || true
printf '%s\n' '== Router key and payload construction =='
ast-grep outline scripts/ci/agent_mention_router.py --items all 2>/dev/null || true
rg -n -C 15 \
'def agent_invocation_key|def noema_payload|def opencode_payload|base_branch|invocation_key' \
scripts/ci/agent_mention_router.py
printf '%s\n' '== Wrapper validation, forwarding, and controls =='
for file in \
.github/workflows/agent-mention-noema-dispatch.yml \
.github/workflows/agent-mention-opencode-dispatch.yml \
.github/workflows/agent-mention-router.yml \
.github/workflows/agent-mention-router-quality-ci.yml
do
if [ -f "$file" ]; then
printf '\n-- %s --\n' "$file"
cat -n "$file"
fi
done
printf '%s\n' '== Invocation regression tests =='
fd -t f . tests 2>/dev/null |
rg 'agent_mention' |
while IFS= read -r file; do
printf '\n-- %s --\n' "$file"
rg -n -C 8 \
'(invocation.key|base.branch|sha256|digest|wrapper|payload|ledger|partial|budget|acknowledg)' \
"$file" || true
done
printf '%s\n' '== Exact-head check and review state =='
gh pr view 787 --repo "$repo" \
--json headRefOid,baseRefOid,state,isDraft,mergeStateStatus,reviewDecision,reviews,statusCheckRollup \
--jq '{headRefOid,baseRefOid,state,isDraft,mergeStateStatus,reviewDecision,reviews:[.reviews[]|{author:.author.login,state,commitId:.commit.oid,submittedAt}],checks:[.statusCheckRollup[]|{name,status,conclusion,headSha}]}'
printf '%s\n' '== Failed, cancelled, or incomplete exact-head checks =='
gh api "repos/${repo}/commits/${head}/check-runs?per_page=100" \
--jq '.check_runs[] | select(.conclusion != "success" and .conclusion != "skipped" and .conclusion != "neutral") | {id,name,status,conclusion,details_url,output}'
printf '%s\n' '== Branch protection =='
gh api "repos/${repo}/branches/main/protection" \
--jq '{required_pull_request_reviews,required_status_checks,enforce_admins,required_conversation_resolution}' || true
printf '%s\n' '== Unresolved current review threads =='
gh api graphql -F owner="ContextualWisdomLab" -F name=".github" -F number=787 -f query='
query($owner: String!, $name: String!, $number: Int!) {
repository(owner: $owner, name: $name) {
pullRequest(number: $number) {
reviewThreads(first: 100) {
nodes {
isResolved
isOutdated
path
line
comments(first: 10) { nodes { author { login } body } }
}
}
}
}
}' --jq '.data.repository.pullRequest.reviewThreads.nodes[] | select(.isResolved == false and .isOutdated == false) | {path,line,comments:[.comments.nodes[]|{author:.author.login,body}]}'Length of output: 78480 🏁 Script executed: #!/bin/bash
set -euo pipefail
printf '%s\n' '== Router dispatch and acknowledgement boundaries =='
sed -n '245,455p' scripts/ci/agent_mention_router.py | cat -n
printf '%s\n' '== Sweep inventory, dispatch budget, and credential source handling =='
ast-grep outline scripts/ci/agent_mention_sweep.py --items all 2>/dev/null || true
sed -n '1,460p' scripts/ci/agent_mention_sweep.py | cat -n
printf '%s\n' '== Repository-dispatch consumers and review-only controls =='
rg -n -i -C 8 \
'(repository_dispatch:|types:\s*\[?(noema-review|merge-scheduler)|event_type:\s*"(noema-review|merge-scheduler)"|agent_invocation_key|enable_auto_merge|update_branches|merge_mode|pulls/.*/merge|releases|branch protection)' \
.github/workflows scripts/ci
printf '%s\n' '== Full workflow-contract test =='
cat -n tests/test_agent_mention_workflow_contract.py
printf '%s\n' '== Targeted key, Noema base-branch, and sweep tests =='
rg -n -C 10 \
'(base_branch|invocation_key|canonical|digest|partial|budget|credential|fail.closed|repository_source|workflow.run)' \
tests/test_agent_mention_idempotency.py \
tests/test_agent_mention_sweep.py \
tests/test_agent_mention_downstream_idempotency.py \
tests/test_agent_mention_receipt_authority.py
printf '%s\n' '== Mutation-capable GitHub API calls in changed automation =='
rg -n -i -C 4 \
'(gh api .* -X (POST|PUT|PATCH|DELETE)|/dispatches|/comments|/reactions|/merge|/releases|branches/.*/protection)' \
scripts/ci .github/workflows/agent-mention-*.yml
printf '%s\n' '== Exact-head check workflow runs and annotations for cancelled runs =='
for run_id in 31077409462 31077407485 31077816786; do
printf '\n-- workflow run %s --\n' "$run_id"
gh run view "$run_id" --repo ContextualWisdomLab/.github \
--json databaseId,headSha,status,conclusion,event,workflowName,jobs \
--jq '{databaseId,headSha,status,conclusion,event,workflowName,jobs:[.jobs[]|{name,status,conclusion,startedAt,completedAt}]}'
doneLength of output: 246652
The prior wrapper finding is resolved.
I found no new source-level blocker in the reviewed mention-router boundary. I cannot approve this exact head yet.
Wait for all exact-head required checks to complete successfully. Then obtain a qualifying independent current-head approval and verify branch protection with a credential that can read it. You are interacting with an AI system. |
|
Exact-head review request for @coderabbitai review |
|
I will use current-head evidence only. I will verify the wrapper key binding,
|
There was a problem hiding this comment.
Pull request overview
OpenCode cannot approve yet because required coverage evidence did not pass.
Review outcome
1. HIGH .github/workflows/opencode-review.yml:1 - Coverage evidence did not prove required test/docstring evidence
-
Problem: The required coverage-evidence job result was
failure, so OpenCode cannot establish approval sufficiency for this head. -
Root cause: Automated approval is only valid when the same-head coverage-evidence job proves supported repository test suites passed and configured docstring gates passed or were advisory, or reports not applicable because no supported source files or package manifests exist. Missing, failed, skipped, unavailable, or unsupported-tooling test evidence is a blocker.
-
Fix: Install or configure the repository test/docstring evidence tooling when source files or package manifests exist, rerun the current-head coverage-evidence job, and approve only after it reports
successwith required evidence or explicit no-source not-applicable evidence. -
Regression test: Keep the approval branch checking
needs.coverage-evidence.result == successbefore posting APPROVE, and publish REQUEST_CHANGES when coverage-evidence blocker states such as cancelled, skipped, failed, unsupported-tooling, or below-100 evidence are present. -
Result: REQUEST_CHANGES
-
Reason: coverage-evidence result was
failure, so required test/docstring evidence was not proven for current head92c83f44c75cae42e90dc6984dc89c696b6c3db9. -
Head SHA:
92c83f44c75cae42e90dc6984dc89c696b6c3db9 -
Workflow run: 31078896274
-
Workflow attempt: 1
Coverage evidence
Coverage Decision
- Result: FAIL
- Test evidence: not proven passing
- Docstring evidence: not proven passing when configured
- Failure count: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Workflow (4 files)"]
S1 --> I1["GitHub Actions review job"]
I1 --> R1["Review risk: Workflow (4 files)"]
R1 --> V1["actionlint plus required checks"]
Evidence --> S2["Changed file: CHANGELOG.md"]
S2 --> I2["repository behavior"]
I2 --> R2["Review risk: Changed file: CHANGELOG.md"]
R2 --> V2["required checks"]
Evidence --> S3["Docs: review-agent-comment-invocation.md"]
S3 --> I3["operator or user guidance"]
I3 --> R3["Review risk: Docs: review-agent-comment-invocation.md"]
R3 --> V3["docs review"]
Evidence --> S4["CI script (2 files)"]
S4 --> I4["review and security gate shell path"]
I4 --> R4["Review risk: CI script (2 files)"]
R4 --> V4["bash -n plus Strix self-test"]
Evidence --> S5["Test (7 files)"]
S5 --> I5["regression suite"]
I5 --> R5["Review risk: Test (7 files)"]
R5 --> V5["targeted test run"]
seonghobae
left a comment
seonghobae
left a comment
There was a problem hiding this comment.
Exact current head 92c83f44c75cae42e90dc6984dc89c696b6c3db9 is reduced to the durable 15-file router/runtime/test/documentation scope; every PR-specific repair workflow, trigger, and patch helper has been removed. Agent Mention Router Quality CI, Python Security, CodeQL, Security Scan, Semgrep, Secret Scan, OSV, Scorecard, SBOM, and CodeRabbit status all succeeded; all inline threads are resolved. Requesting fresh exact-head independent review without branch mutation, author approval, or merge bypass.
seonghobae
left a comment
seonghobae
left a comment
There was a problem hiding this comment.
Review-only exact-head request for 92c83f44c75cae42e90dc6984dc89c696b6c3db9; do not update branches, approve from author identity, or merge. Validate exact-key distributed idempotency, per-agent partial completion, durable wrapper-run authority, bounded dispatch accounting, review-only mutation controls, 100% production statement/branch coverage, and public docstrings.
seonghobae
left a comment
seonghobae
left a comment
There was a problem hiding this comment.
@cwl-noema-review
Independently review exact head 92c83f44c75cae42e90dc6984dc89c696b6c3db9, focusing on invocation authority, partial-failure idempotency, sibling-sweep starvation resistance, token boundaries, and absence of branch mutation or merge authority.
2 participants
Purpose
Provide a central, bounded comment-invocation surface for the existing Noema and OpenCode review planes without copying privileged implementation into product repositories.
Product behavior
@cwl-noema-reviewand review-only@opencode-agentcomments;OWNER,MEMBER, orCOLLABORATORidentities and reject bot-authored requests;issue_commentevents into their default branches;enable_auto_merge=false,update_branches=false, andmerge_mode=disabledcontrols;workflow_dispatchentrypoint remains;ubuntu-24.04runners and immutableactions/checkoutv7.0.1 pins;contents: writeonly to jobs and wrapper workflows that call GitHub's create-repository-dispatch endpoint, while workflow defaults remain read-only and the existing review-agent credential chain remains unchanged;Reliability and security repairs
Review-driven regressions now prove the complete distributed idempotency boundary:
Exact-head verification
Current exact head:
8aa7101fdb6a9993e1699639362d6612f6cc42c5.Current protected base:
f070c504c1cb06891b800d7ab0cf6ac7d3cf8eae.Successful on this exact head:
31063991508;31063991507;31063991539;31063991515;31063991514;31063991505;31063991708;31063991525;31063991502;The predecessor-head OpenCode request-changes review for
ff02c877db5fbe2863f46b9639362d6612f6cc42c5does not exist; the dismissed stale review was anchored toff02c877db5fbe2863f46b963b1d5c27b5acae37and its failed coverage run. It was dismissed only after the current exact head passed the complete focused and security gates. A new exact-current-head CodeRabbit, OpenCode, Noema, and qualifying non-author verdict remains required.Integration boundary
The router is a control-plane invocation surface only. Existing Noema and OpenCode workflows retain reviewer identity, credential resolution, model policy, findings, approvals, and merge policy. Wrapper workflows forward each deterministic exact-key invocation at most once to those authoritative planes. Standalone product repositories require no copied privileged implementation.
Merge acceptance
Do not merge until a qualifying independent non-author approval is anchored to exact head
8aa7101fdb6a9993e1699639362d6612f6cc42c5and branch protection permits merge without administrative bypass. Prior-head reviews, author comments, stale requested-changes verdicts, successful checks without a current verdict, rate-limit notices, target-repository acknowledgements, and reactions are not approval evidence.Supersedes the unique mention-router slice of closed predecessor #757; no predecessor approval or check is reused.